Cache for git remote by jrd · Pull Request #1882 · firecow/gitlab-ci-local

jrd · 2026-06-15T21:10:58Z

Based on #1836 it allows to keep some cache for remote refs so it’s a bit faster.

Summary by cubic

Speeds up include processing by caching identical git ls-remote calls and preventing races by writing the component parse cache only after remote downloads finish.

Performance
- Add in-memory cache for git ls-remote results keyed by the full command; cleared via ParserIncludes.resetCount().
- Defer componentParseCache.set until after remote component download completes to avoid parallel resolution races.
Refactors
- Replace exported getGitRemoteInfo with ParserIncludes.getGitRemoteInfo and update call sites.

^{Written for commit eeea380. Summary will update on new commits.}

cubic-dev-ai

2 issues found across 9 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

firecow · 2026-06-22T11:11:20Z

I think #1836 already covers the component.* side nicely, so I'm going to close this one to keep things simple.
If performance issues arise, we can revisit later.

jrd · 2026-06-22T11:17:58Z

Well @firecow . Here is one of my .gitlab-ci.yml:

include:
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/security@0.8
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/python-linter@0.8
    inputs:
      system-packages: 'libgc++'
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/python-i18n@0.8
    inputs:
      system-packages: 'libgc++'
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/python-tests@0.8
    inputs:
      system-packages: 'libgc++'
      future-enable: true
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/js-linter@0.8
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/js-i18n@0.8
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/js-tests@0.8
    inputs:
      future-enable: true
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/total-coverage@0.8
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/docker-test@0.8
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/tag@0.8
    inputs:
      needs:
        - security
        - secret
        - python-linter
        - python-i18n
        - python-tests
        - js-linter
        - js-i18n
        - js-tests
        - docker-test
  - component: $CI_SERVER_FQDN/systracorp/hq/infra/ci/docker-publish@0.8

# extra parameters to use a postgresql service
python-tests: &python-pg-svc
  services:
    - name: postgres:17-alpine
      alias: db
      variables:
        POSTGRES_DB: syspro
        POSTGRES_USER: syspro
        POSTGRES_PASSWORD: syspro
  variables:
    DB_HOST: db
python-tests:future:
  <<: *python-pg-svc

Without this patch, it takes 19 seconds to run gitlab-ci-local --list.

With it, 3 seconds.

This is with the cache already there for both.

Multiple component at the same version from the same catalog is the reason this takes a long time without this optimization.

firecow · 2026-06-22T11:20:58Z

Holy christ 😄

Ok, I'm reopening. Jesus, what a pipeline.

firecow · 2026-06-22T11:46:30Z

@jrd rebase onto master now that #1836 landed

firecow · 2026-07-31T07:02:27Z

+                promises.push((async (componentParseCache: Map<number, ParsedComponent>, component: ParsedComponent, opts: ParserIncludesInitOptions) => {
+                    if (!component.isLocal) {
+                        await this.downloadIncludeComponent(opts, component.projectPath, component.effectiveRef, component.componentPath);
+                    }
+                    componentParseCache.set(index, component);
+                })(componentParseCache, component, opts));
            }


Suggested change

promises.push((async (componentParseCache: Map<number, ParsedComponent>, component: ParsedComponent, opts: ParserIncludesInitOptions) => {

if (!component.isLocal) {

await this.downloadIncludeComponent(opts, component.projectPath, component.effectiveRef, component.componentPath);

}

componentParseCache.set(index, component);

})(componentParseCache, component, opts));

}

promises.push((async () => {

if (!component.isLocal) {

await this.downloadIncludeComponent(opts, component.projectPath, component.effectiveRef, component.componentPath);

}

componentParseCache.set(index, component);

})());

firecow · 2026-07-31T07:02:27Z

+        const cmdStr = cmdArgs.join(" ");
+        let info = this.gitRemoteInfoCache[cmdStr];
+        if (!(cmdStr in this.gitRemoteInfoCache)) {
+            info = Utils.syncSpawn(cmdArgs).stdout;
+            this.gitRemoteInfoCache[cmdStr] = info;
+        }
+        return info;
+    };


Suggested change

const cmdStr = cmdArgs.join(" ");

let info = this.gitRemoteInfoCache[cmdStr];

if (!(cmdStr in this.gitRemoteInfoCache)) {

info = Utils.syncSpawn(cmdArgs).stdout;

this.gitRemoteInfoCache[cmdStr] = info;

}

return info;

};

const cmdStr = cmdArgs.join(" ");

if (!(cmdStr in this.gitRemoteInfoCache)) {

this.gitRemoteInfoCache[cmdStr] = Utils.syncSpawn(cmdArgs).stdout;

}

return this.gitRemoteInfoCache[cmdStr];

}

firecow · 2026-07-31T07:02:27Z

        if (gitRemoteMatch?.groups == null) throw new Error(`This is a bug, please create a github issue if this is something you're expecting to work. input: ${component}`);
        const {domain, projectPath, port, componentName, ref} = gitRemoteMatch.groups;
        const isLocalComponent = projectPath === `${gitData.remote.group}/${gitData.remote.project}` && ref === gitData.commit.SHA;
+        const parserIncludes = this; // eslint-disable-line @typescript-eslint/no-this-alias


Suggested change

const parserIncludes = this; // eslint-disable-line @typescript-eslint/no-this-alias

firecow · 2026-07-31T07:02:27Z

                        if (this.reference == "~latest" || semanticVersionRangesPattern.test(this.reference)) {
                            // https://docs.gitlab.com/ci/components/#semantic-version-ranges
-                            const stdout = getGitRemoteInfo(this, "--tags");
+                            const stdout = parserIncludes.getGitRemoteInfo(this, "--tags");


Suggested change

const stdout = parserIncludes.getGitRemoteInfo(this, "--tags");

const stdout = ParserIncludes.getGitRemoteInfo(this, "--tags");

firecow · 2026-07-31T07:02:27Z

                        this._cache.sha = this.effectiveRef;
                    } else {
-                        const stdout = getGitRemoteInfo(this);
+                        const stdout = parserIncludes.getGitRemoteInfo(this);


Suggested change

const stdout = parserIncludes.getGitRemoteInfo(this);

const stdout = ParserIncludes.getGitRemoteInfo(this);

firecow

The 5 suggestions must be batched. Deleting the alias line alone breaks the build; lines 300 and 326 still reference parserIncludes.

Also: the diff still contains the component.* work #1836 superseded; the rebase you asked for should reduce it to just the caching change.

firecow · 2026-07-31T07:19:14Z


 export class ParserIncludes {
    private static count: number = 0;
+    private static gitRemoteInfoCache: Record<string, string> = {};


Never cleared, though resetCount() sits right beside it for count. Stale git ls-remote results could leak between runs in the test suite. Broke nothing I ran, so your call whether it matters for a single-shot CLI.

Yes I can reinit the remote info cache in the resetCount method. That would be good enough I think.

jrd · 2026-08-06T12:15:28Z

Should be good now.

cubic-dev-ai Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread src/parser.ts

Comment thread src/parser-includes.ts

jrd force-pushed the cache_for_git_remote branch from f5279b8 to cf7ad82 Compare June 22, 2026 11:10

firecow closed this Jun 22, 2026

firecow reopened this Jun 22, 2026

firecow reviewed Jun 22, 2026

View reviewed changes

Comment thread src/parser-includes.ts Outdated

jrd force-pushed the cache_for_git_remote branch from 5652d2b to 1c3a1fd Compare June 25, 2026 10:54

jrd requested a review from firecow June 30, 2026 23:53

firecow reviewed Jul 31, 2026

View reviewed changes

Use some in-memory cache for git ls-remote identical calls

eeea380

jrd force-pushed the cache_for_git_remote branch from 1c3a1fd to eeea380 Compare August 5, 2026 15:49

	const stdout = parserIncludes.getGitRemoteInfo(this, "--tags");
	const stdout = ParserIncludes.getGitRemoteInfo(this, "--tags");

Conversation

jrd commented Jun 15, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

firecow commented Jun 22, 2026

Uh oh!

jrd commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

firecow commented Jun 22, 2026

Uh oh!

firecow commented Jun 22, 2026

Uh oh!

Uh oh!

firecow Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

firecow Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

firecow Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

firecow Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

firecow Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

firecow left a comment

Choose a reason for hiding this comment

Uh oh!

firecow Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrd Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

jrd commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jrd commented Jun 15, 2026 •

edited by cubic-dev-ai Bot

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

jrd commented Jun 22, 2026 •

edited

Loading

firecow Jul 31, 2026 •

edited

Loading